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Abstract 


Analyses  of  alternatives  (AoAs)  play  an  important  part  in  the  acquisition 
process.  The  act  of  selecting  a  set  of  alternatives  to  be  compared  in  an  AoA  can 
help  decision  makers  understand  and  manage  key  tradeoffs — or  it  can  prematurely 
constrict  or  otherwise  distort  the  solution  trade  space.  It  is  fairly  easy  to  point  to 
completed  AoAs  as  evidence  that  many  of  them  have  been  poorly  scoped.  But  it  is 
much  more  difficult  to  spot  the  problem  in  real  time — or  to  keep  it  from  occurring  at 
all. 

This  report  identifies  four  principles  designed  to  help  minimize  the  occurrence 
of  poorly  scoped  AoAs.  These  four  principles  were  arrived  at  by  applying  formalisms 
from  the  disciplines  of  systems  analysis  and  systems  thinking,  in  combination  with  a 
series  of  semi-structured  interviews  with  members  of  the  AoA  stakeholder 
community  (consumers,  sponsors,  practitioners,  and  critics). 

The  principles  may  be  summarized,  in  systems  terminology,  as  Focus  on 
Outputs  and  Think  Backwards;  Start  With  the  Exterior  and  Work  Inwards;  Apply 
Constraints  Carefully;  and  Iterate  and  Reduce  Uncertainty.  The  report  translates 
these  principles  into  practical  terms  that  can  be  understood  and  applied  by  AoA 
stakeholders.1 

Keywords:  Analysis  of  Alternatives,  AoA,  systems  analysis,  system 
boundary,  systems  thinking,  trade  space,  Weapon  Systems  Acquisition  Reform  Act, 
WSARA 


1  This  report  was  developed  under  work  supported  by  the  Naval  Postgraduate  School  Acquisition 
Research  Program  Assistance  Agreement  No.  N00244-1 3-1-0036  awarded  by  the  Naval  Supply 
Systems  Command  (NAVSUP)  Fleet  Logistics  Center  (FLC)  San  Diego.  It  has  not  been  formally 
reviewed  by  NPS.  The  views  and  conclusions  contained  in  this  document  are  those  of  the  authors 
and  should  not  be  interpreted  as  necessarily  representing  the  official  policies,  either  expressed  or 
implied,  of  the  NPS  or  NAVSUP  FLC  San  Diego.  The  NPS  and  NAVSUP  FLC  San  Diego  do  not 
endorse  any  products  or  commercial  services  mentioned  in  this  publication. 
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Analyses  of  Alternatives  (AoAs):  Toward  a 
More  Rigorous  Determination  of  Scope 


Introduction 

Background 

Analysis  of  Alternatives 

Within  the  framework  of  the  defense  acquisition  system  (DAS),  an  analysis  of 
alternatives  (AoA)  “assesses]  the  potential  materiel  solutions  to  satisfy  the  capability 
need  documented  in  [an]  approved”  initial  capabilities  document  (ICD).2  Conducted 
ad  hoc  during  the  1970s  and  1980s  under  the  label  of  cost  and  operational 
effectiveness  analysis  (COEA),  this  type  of  assessment  has  been  an  important 
feature  of  the  acquisition  landscape  ever  since  a  major  revision  in  the  DoD  5000- 
series  regulations  in  1991. 3 

Arguably,  this  emphasis  has  proved  to  be  a  mixed  blessing  over  the  years. 

On  the  one  hand,  the  “formality  and  inevitability”  of  AoAs  has  helped  guard  against 
the  potential  for  premature  selection  of  a  preferred  alternative.4  However,  the  very 
complexity  of  the  process  has  also  made  it  prone  to  start-up  delays,  which,  when 
combined  with  the  pressure  for  timely  decisions,  leads  to  the  seemingly  inevitable 
result  that  “the  time  for  actual  analysis  compresses.”5  Something  has  to  give,  and  a 
common  practice  has  been  to  narrow  the  AoA  scope  (for  example,  by  constraining 
the  set  of  alternatives). 

Complicating  the  problem  of  AoA  scoping  has  been  a  steady  pressure  to 
make  the  rigorous  comparison  of  alternative  solutions  a  mandatory  step  that  occurs 
earlier  and  earlier  in  the  acquisition  timeline.  The  1996  revision  of  the  DoD  5000 
series  formalized  the  requirement  for  AoAs  to  be  performed  before  Milestone  B 
(program  initiation).6  And  the  2003  revision — which  was  accompanied  by  the 
introduction  of  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS) — 
included,  for  the  first  time,  a  requirement  for  AoAs  to  be  conducted  during  the 


2  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  [USD(AT&L)],  2008, 
Enclosure  2,  section  4.C.5. 

3  Under  Secretary  of  Defense  for  Acquisition  (USD[A])  (1 991 )  contained  detailed  guidance  for 
conducting  and  documenting  these  analyses.  Balut  et  al.  (2004),  p.  19,  presented  a  chronology  from 
a  cost  analysis  perspective  and  emphasized  the  importance  of  the  1991  revision. 

4  Smith  &  Thompson,  1 995,  p.  1 1 . 

5  Ibid.,  p.  14. 

6  Under  Secretary  of  Defense  for  Acquisition  and  Technology  (USD[A&T]),  1996a,  1996b. 
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Materiel  Solution  Analysis  phase,  that  is,  prior  to  Milestone  A.7 * 9  The  earlier  the  AoA, 
the  wider  the  set  of  potential  alternatives  (see  Figure  1 )  and  the  more  disparate  they 
are.  As  a  result,  it  is  more  difficult  to  decide  which  of  them  should  be  considered  in 
the  assessment. 


Figure  1.  AoA  Scoping  in  the  Context  of  DAS  Phases  and  Milestones 
Evidence  of  a  Problem 

Unfortunately,  there  is  evidence  that  the  AoAs  have  not  always  been  properly 
scoped.  In  2009,  the  Government  Accountability  Office  (GAO)  reviewed  32  major 
DoD  acquisition  programs,  22  of  which  had  included  a  formal  AoA.  Of  these,  13 
AoAs — more  than  half — were  characterized  as  overly  “narrow  [in]  scope”  in  the 
sense  that  they  “focused  on  a  limited  number  of  alternatives.”8  Moreover,  there  was 
a  strong  correlation  between  AoAs  with  a  narrowly  scoped  set  of  alternatives  and  the 
occurrence  of  cost  growth  in  the  ensuing  programs.9 

More  anecdotally,  it  has  been  common  in  our  experience  to  hear  AoAs 
described  as  being  scoped  around  a  predetermined  solution.  (“By  the  time  you  reach 
the  Materiel  Development  Decision  [MDD],  someone  has  already  decided  what  the 
solution  is  going  to  be.”)10  To  the  extent  this  is  true,  the  AoA  becomes  a  square¬ 
filling  exercise  or,  at  best,  a  process  of  exploring  minor  embellishments  around  a 


7  USD(AT&L),  2003a,  2003b.  The  requirement  for  functional  area,  functional  need,  and  functional 

solution  analysis  spelled  out  in  the  CJCS  3170  series  of  instructions  in  2003  can  be  interpreted  as  a 
desire  to  push  this  process  even  further  to  the  left,  so  to  speak,  on  the  timeline.  Interestingly,  Smith 
and  Thompson  had  eight  years  earlier  argued  the  case  for  “putting]  the  initial  [AoA]  into  the  time 
normally  reserved  for  ‘requirements  analysis’” — or,  at  the  least,  “importing]  some  of  the  attributes  of 
[an  AoA]  into  the  requirements  analysis  process  that  already  takes  place”  (p.  15). 
feGAO,  2009,  pp.  7,  9. 

9  Ibid.,  pp.  10-12. 

10  See  also  Husband  and  Kaspersen  (2012),  p.  9;  Smith  &  Thompson  (1995),  p.  11. 
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particular  type  of  technology,  weapon,  platform,  or  piece  of  equipment.  The  potential 
to  conduct  a  wide-open  look  at  alternative  technology  solutions  is  lost. 

2009  WSARA  and  Other  Reforms 

The  Weapon  Systems  Acquisition  Reform  Act  (WSARA)  of  2009  included 
important  changes  designed  in  part  to  address  this  problem.  A  major  change  in 
governance  was  the  designation  of  the  Director,  Cost  Assessment  and  Program 
Evaluation  (DCAPE) — a  Senate-confirmed  position — as  responsible  not  only  for 
formulating  AoA  guidance  but  also  for  the  performance  of  the  analysis.  This 
responsibility  includes  the  authority  to  reject  or  redirect  an  AoA. 

WSARA  specifies  that  the  AoA  study  guidance  promulgated  by  DCAPE 
should  ensure  the  “full  consideration  of  possible  trade-offs  among  cost,  schedule, 
and  performance  objectives.”11  By  implication,  “proper  consideration  of  tradeoffs 
includes  ensuring  ...  that  a  range  of  sufficiently  different  alternatives  are 
examined.”12 

To  help  discharge  these  responsibilities  during  AoA  execution,  CAPE  typically 
establishes  a  senior  advisory  group  (SAG)  that  includes  the  Office  of  the  Under 
Secretary  for  Acquisition,  Technology,  and  Logistics  (AT&L)  as  well  as  Joint  Staff 
and  service  representatives.  In  theory,  the  SAG  provides  a  mechanism  for  users, 
analysts,  and  overseers  to  adjust  AoA  scope  during  execution  if  necessary. 

Implementation  of  WSARA  has  been  accompanied  by  a  push  to  shorten  AoA 
timelines.  The  typical  duration  of  a  pre-WSARA  AoA  has  been  variously  estimated  at 
16  to  24  months.13  In  contrast,  several  stakeholders  interviewed  for  this  project  cited 
6  to  9  months  as  the  current  goal.  This  shortening  of  AoA  timelines  will  have 
significant  impacts — both  positive  and  negative — on  the  problem  of  AoA  scope; 
these  are  discussed  further  in  the  section  titled  Why  are  Some  AoAs  Poorly 
Scoped? 

The  number  of  completed  AoAs  conducted  entirely  within  the  post-WSARA 
timeframe  is  modest.  The  consensus  within  the  acquisition  community  seems  to  be 
that  it  is  still  too  soon  to  say  exactly  how  the  formal  realignment  of  AoA 
responsibilities,  the  introduction  of  the  SAG,  and  the  push  for  dramatically  shorter 
timelines  have  impacted  AoA  scoping. 


11  111th  Congress  of  the  United  States,  Public  Law  111-23,  §  201(d)(1). 

12  Husband  and  Kaspersen,  2012,  p.  9. 

13  Kowal,  in  2009,  gave  an  average  of  two  years  with  up  to  six  years  in  some  cases.  At  the  other 
extreme,  information  obtained  from  stakeholder  interviews  cited  an  average  of  16  months.  By  way  of 
corroboration,  Lihani  (2011)  cited  USAF  Office  of  Aerospace  Studies  data  on  27  AoAs  conducted 
from  2000  to  2008;  the  average  length  was  20  months.  Of  the  22  AoAs  considered  in  GAO  (2009), 
the  majority  took  between  13  and  30  months. 
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Purpose 

This  project  aimed  to  identify  a  set  of  guiding  principles  that  can  be  applied  by 
AoA  practitioners  and  sponsors  to  reduce  the  incidence  of  improperly  scoped 
analyses.  In  particular,  we  wanted  to  develop  principles  that  were  both  (a)  grounded 
in  the  discipline  of  systems  thinking14  and  (b)  practical,  in  the  sense  that  they  can  be 
readily  understood  and  applied. 

In  striving  for  more  rigor,  we  are  not  proposing  to  remove  all  subjectivity  or 
sense  of  “art”  from  the  job  of  scoping  an  AoA.  Rather,  we  sought  a  middle  ground,  in 
which  decisions  regarding  AoA  scope  can  be  influenced  by  considerations  more 
systematic  and  rigorous  than  those  arising  solely  from  some  combination  of  political, 
bureaucratic,  and  programmatic  pressures. 

Limitations 

This  paper  is  not  intended  as  a  guide  to  conducting  AoAs  or  related  studies. 
Many  such  guides  do  exist,  and  the  findings  and  recommendations  presented  here 
are  meant  to  be  understood  and  applied  in  the  context  of  those  references.15 
Similarly,  it  is  assumed  that  the  reader  already  has  a  basic  familiarity  with  the  DAS 
and  JCIDS  processes. 

In  this  paper,  the  term  scope  is  associated  with  the  set  of  alternatives.  To  be 
sure,  there  are  other  dimensions  of  scope,  such  as  the  set  of  scenarios, 
assumptions,  or  study  constraints.  We  considered  these  other  dimensions  only 
insofar  as  they  shape  decisions  about  the  set  of  alternatives. 

Factors  such  as  organizational  roles  and  responsibilities,  institutional  culture, 
and  even  the  predilections  of  individual  decision  makers  and  executives  all 
contribute  to  the  incidence  of  improper  AoA  scoping.  While  acknowledging  the 
importance  of  such  factors,  this  paper’s  recommendations  do  not  address  them 
directly. 

A  Note  on  Terminology 

The  word  system  can  have  many  connotations.  (Within  the  opening  pages  of 
this  paper,  for  instance,  we  have  already  referred  to  the  acquisition  system,  the  Joint 
Capabilities  Integration  and  Development  System,  and  the  weapon  systems  that 
may  represent  alternative  solutions  to  a  capability  need.)  For  clarity,  the  text 
indicates  the  specific  type  of  system  that  is  meant  in  each  case.  In  particular,  the 
ships,  aircraft,  vehicles,  sensors,  communications  equipment,  computers,  and  so 


14  See  Edson  (2008). 

15  For  example,  see  USAF  Office  of  Aerospace  Studies  (2010);  Joint  Staff  (2009);  and  Defense 
Acquisition  University  (2013),  section  3.3  (Analysis  of  Alternatives). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  4  - 

NAVAL  POSTGRADUATE  SCHOOL 


forth  that  may  be  the  objects  of  study  in  an  AoA  are  collectively  referred  to  as 
combat  or  combat  support  systems  (C/CSSs). 

Approach 

The  first  step  in  applying  a  systems  thinking  approach  is  to  understand  the 
nature  of  the  problem.  Only  then  can  we  explore  various  system  views  and  formulate 
the  resulting  insights  as  a  set  of  recommendations  aimed  at  improving  the  practice 
of  determining  AoA  scope.  The  remaining  major  sections  of  the  paper  (titled 
Understanding  the  Problem,  Applying  a  Systems  View,  and  Recommendations  and 
Observations)  follow  this  general  outline. 

This  approach  entailed  two  broad  methods  of  inquiry  and  analysis.  The  first 
method  was  to  leverage  the  existing  body  of  knowledge  by  reviewing  available 
documents  (DoD  and  service  instructions,  guidance,  and  handbooks;  published 
reports  and  articles;  and  other  publicly  available  materials)  and  conducting 
interviews  with  stakeholders  representing  various  roles  (that  is,  identifying  the 
capability  needs  being  addressed  in  AoAs;  overseeing  the  conduct  of  AoAs; 
performing  AoAs;  and  reviewing  and  critiquing  completed  AoAs).  The  second 
method  was  to  apply  formal  systems  thinking  paradigms.  These  two  methods  were 
applied  in  concert  and  iteratively:  More  than  once,  we  used  systems  thinking 
formalisms  to  gain  insights  into  an  issue  identified  during  interviews  and  then  used 
subsequent  interviews  to  test  the  validity  of  those  insights. 

All  interviews  were  conducted  on  a  strict  non-attribution  basis.  Statements 
derived  from  interviews  are  substantiated  in  this  report  only  to  the  extent  of 
specifying  that  a  particular  view  was  expressed  by  one,  some,  several,  most,  or  all 
interviewees. 

The  research  and  analysis  presented  herein  was  conducted  from  May  2013 
through  January  2014. 

Understanding  the  Problem 

So  far,  we  have  not  specified  exactly  what  it  means  to  say  that  an  AoA  is 
improperly  scoped.  The  following  section  addresses  this  key  point.  We  then  present 
some  important  causes  of  improper  AoA  scoping,  as  identified  during  stakeholder 
interviews. 

What  Is  a  “Poorly  Scoped”  AoA? 

Establishing  consensus  on  this  question  proved  surprisingly  difficult.  Almost 
all  interviewees,  when  asked  the  question  directly,  preferred  to  skip  over  it  and 
proceed  straight  to  a  discussion  of  causes.  In  some  cases,  that  discussion  made  it 
clear  how  the  interviewee  was  defining  the  problem;  in  other  cases,  it  did  not. 
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Guided  as  we  were  by  a  systems  viewpoint — including  the  notion  that 
alternatives  represent  a  set  that  has  a  boundary  (see  our  discussion  of  the 
Conceptagon  in  a  later  section) — we  consolidated  interviewees’  implied  constructs  in 
the  form  of  two  definitions  based  on  the  properties  of  boundaries.  First,  an  AoA 
could  be  too  narrowly  scoped  if  its  boundary  excludes  alternatives  that  should  be 
considered.  Second,  an  AoA  could  be  too  broadly  scoped  if  it  includes  alternatives 
that  should  not  be  considered.16 

These  definitions,  although  helpful,  still  contain  more  than  a  little  question 
begging.  On  what  basis  can  we  say  that  an  alternative  should  or  should  not  be 
inside  the  AoA  boundary?  Here,  it  is  important  to  remember  that  the  purpose  of  an 
AoA  is  to  inform  a  decision  about  the  materiel  solution  to  a  capability  need.  Thus, 
poor  scoping  is  that  which  contributes  to  a  poor  decision  (for  example,  by  excluding 
a  viable  alternative  that  might  have  been  judged  preferable  under  some  reasonable 
weighing  of  costs  and  benefits)17  or  otherwise  impairs  decision  making  (for  example, 
by  presenting  so  much  information  on  so  many  alternatives  that  it  is  difficult  to  make 
a  good  choice). 

The  two  types  of  improper  scoping  are  further  discussed  in  the  next  two 
sections. 

Too  Narrowly  Scoped 

Here,  one  or  more  viable  alternatives  have  been  excluded,  inappropriately 
limiting  the  solution  trade  space.  The  most  obvious  way  to  detect  this  problem  is 
simply  to  identify  one  or  more  of  the  missing  alternatives.  If  the  AoA  has  not 
provided  a  rationale  for  their  exclusion  or  has  simply  assumed  them  out  of  the 
picture,  we  can  safely  say  that  the  AoA  was  not  well  scoped.  If  a  rationale  is 
provided,  the  test  is  more  difficult:  namely,  to  demonstrate  that  the  alternative  might 
have  been  preferable  under  some  reasonable  weighing  of  costs  and  benefits, 
notwithstanding  the  stated  rationale. 


16  Note  that  under  these  definitions,  a  particular  AoA  could  be  both  too  narrow  and  too  broad  at  the 
same  time!  In  other  words,  it  could  inappropriately  exclude  some  alternatives  and  inappropriately 
include  others. 

17  The  use  of  the  word  reasonable  here  suggests  some  parallel  with  the  “reasonable  person  standard” 
in  tort  law.  Legal  theorists  disagree  over  whether  the  definition  of  “reasonableness”  should  be 
normative  (that  is,  a  reasonable  person  is  one  who  does  what  is  cost-effective,  even  if  most  people 
would  not)  or  positive  (that  is,  a  reasonable  person  is  one  who  does  what  most  people  would  do, 
even  if  it  is  not  cost-effective;  see  Miller  &  Perry,  2012).  Fortunately,  these  distinctions  are  not  so 
pronounced  when  it  comes  to  choosing  an  AoA  alternative.  The  reason  is  that  the  choice  of  a  solution 
path  for  a  DoD  acquisition  is  almost  never  a  completely  objective  determination,  since  it  requires  a 
weighing  of  costs  and  non-monetary  benefits  based  on  decision  makers’  values  and  perspectives. 
Hence,  the  definition  in  the  text  boils  down  to  the  question  of  how  likely  it  is  that  one  could  find  a 
decision  maker  whose  values  and  perspectives  would  have  led  them  to  choose  the  excluded 
alternative. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  6  - 

NAVAL  POSTGRADUATE  SCHOOL 


Note  that  for  an  AoA  in  progress — or  one  that  has  not  yet  begun — avoiding  an 
overly  narrow  scope  requires  a  way  of  projecting  ahead  to  what  the  costs  and 
benefits  might  look  like  at  the  end  of  the  analysis.  This  fact  argues  strongly  for  taking 
an  iterative  approach  to  defining  alternatives,  and  for  explicitly  tracking  the  upper 
and  lower  limits  of  their  likely  costs  and  benefits  at  every  step  along  the  way.  These 
practices  are  reflected  in  the  recommendations  contained  in  the  last  section  of  this 
report.18 

Too  Broadly  Scoped 

Study  teams  operating  under  time  and  resource  constraints  rarely  make  a 
deliberate  choice  to  include  more  alternatives  than  they  can  adequately  analyze 
within  those  constraints.  However,  they  can  miss  opportunities  to  narrow  the  set  of 
alternatives,  thus  making  less  than  optimal  use  of  the  time  and  resources  available. 
Another  possibility  is  that  AoA  guidance  may  prohibit  such  narrowing  by  specifying 
that  certain  alternatives  must  be  considered  in  the  final  presentation  of  study  results. 
Again,  the  opportunity  cost  of  carrying  these  alternatives  all  the  way  through  the 
analysis  may  be  so  great  as  to  impair  the  quality  of  support  ultimately  provided  to  the 
decision  maker. 

Testing  prospectively  (that  is,  before  the  AoA  is  completed)  for  overly  broad 
scoping  is  difficult.  Since  the  range  of  uncertainty  around  the  known  costs  and 
benefits  of  alternatives  generally  starts  out  being  relatively  large,  it  is  easier,  early 
on,  to  make  the  argument  to  leave  an  alternative  in  the  mix  until  the  uncertainties 
can  be  reduced.  (And  thus  easier  to  show  that  an  alternative  has  been  prematurely 
excluded.)  Stated  another  way,  it  takes  a  certain  expenditure  of  analysis  effort  to 
show  that  no  reasonable  decision  maker  would  be  likely  to  prefer  a  given  alternative. 

Why  Are  Some  AoAs  Poorly  Scoped? 

Our  interviews  with  AoA  consumers,  overseers,  practitioners,  and  critics 
disclosed  several  reasons  why  AoAs  may  be  improperly  scoped. 

Inappropriate  Treatment  of  Time  Constraints 

In  the  first  section,  we  noted  the  recent  trend  toward  more  stringent  time 
constraints  on  AoAs.  Such  constraints,  per  se,  are  not  necessarily  a  problem.  In  fact, 
they  can  contribute  to  better  scoping  by  avoiding  the  situation  in  which  a  protracted 
analysis  or  decision-making  process  fails  to  keep  pace  with  fact-of-life  changes  (see 
the  Lack  of  Agility  section).  However,  AoA  stakeholders  may  deal  with  such 


18  They  are  also  consistent  with  recognized  best  practices  for  screening  alternatives.  See,  for 
example,  USAF  Office  of  Aerospace  Studies  (2008),  section  9.1,  which  describes  not  only  initial 
screening  for  nonviable  alternatives,  but  also  “preliminary  screening,”  “later  screening,”  and  “final 
selection.” 
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constraints  by  compressing  the  timeline  in  inappropriate  ways.  This  may  occur 
before,  during,  or  after  the  AoA  itself. 

Some  types  of  compression  occur  well  before  AoA  initiation,  during  the  JCIDS 
process.  For  instance,  an  advocate  of  a  particular  new  capability  may  save  time  by 
pointing  to  a  prior,  approved  ICD  as  inclusive  of  the  unaddressed  need  for  it.  The 
danger  is  that  the  new  capability  in  question  (for  example,  hold  a  particular  type  of 
target  at  risk)  may  include  a  constraint  (for  example,  with  zero  risk  of  collateral 
damage)  that  was  not  analyzed  in  the  original  Capabilities-Based  Analysis  (CBA). 
Since  the  ICD  is  already  approved,  further  analysis  of  this  constraint  is  not 
performed.  As  a  result,  alternatives  that  cause  only  a  little  collateral  damage  are 
guaranteed  to  be  excluded  from  the  trade  space.  In  short,  there  is  no  process  check 
on  a  narrowing  of  alternatives  that  has  occurred  even  before  the  subsequent  AoA 
has  begun. 

Immediately  before  or  during  an  AoA,  the  timeline  can  be  compressed  by 
summarily  excluding  certain  alternatives,  without  adequately  considering  their 
potential  effectiveness,  cost,  risk,  and  feasibility.  This  action  is  qualitatively  different 
from  accelerating  the  rate  at  which  screening  occurs  within  the  analysis  process:  It 
simply  restricts  the  trade  space  based  on  what  is  assumed  to  be  true.19  In  so  doing, 
it  greatly  increases  the  risk  of  an  overly  narrow  AoA. 

Finally,  consider  the  case  in  which  an  AoA  study  team  has  used  a  limited 
amount  of  time  to  identify  a  range  of  costs  and  benefits  for  a  set  of  alternatives. 
Suppose  the  resulting  uncertainties  are  such  that  there  is  considerable  overlap  and 
no  clear  basis  for  a  decision.  If  additional  analysis  could  sufficiently  narrow  the 
uncertainties,  eliminate  some  of  the  alternatives,  and  clarify  the  decision,  then  a 
refusal  to  extend  the  analysis  timeline  may  be  considered  an  inappropriate 
constraint  that  contributed  to  an  overly  broad  scope  for  the  prematurely  finished 
AoA. 

Inappropriate  Focus  on  Existing  C/CSS  and/or  Operations  Concepts 

Most  interviewees  stated  a  view  that  can  be  paraphrased  as  follows: 
“Whenever  I  see  something  titled  ‘[XYZ]  Replacement  AoA,’  I  automatically  question 
whether  the  scope  may  be  too  narrow.”  The  2009  GAO  study  mentioned  earlier 


19  A  recent  example:  Starosta  (2013)  cited  a  senior  Air  Force  official  regarding  the  upcoming  “Ground- 
Based  Strategic  Deterrent  AoA.”  “[The]  team  is  starting.  We  did  get  re-vectored,  though.  The 
department,  in  this  constrained  budget  environment,  would  like  to  do  those  a  little  faster  with  a  little 
less  money,  and  so  they  have  proposed  a  way  to  streamline  how  they’re  going  to  perform  that  study.” 
The  article  goes  on  to  describe  the  proposed  streamlining:  “In  effect,  the  narrower  AoA  will  look  into 
fewer  modernization  concepts  than  originally  planned.”  The  de-scoping  was  reportedly  performed  by 
a  SAG  based  on  “data  derived  from  previous  analyses.”  Note  that  we  are  not  in  a  position  to  judge 
the  merits  of  this  particular  de-scoping  or  to  declare  it  inappropriate.  The  point  is  simply  that  time 
constraints  can  potentially  exacerbate  the  problem  of  AoA  scoping. 
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made  a  similar  point:  An  AoA  for  a  helicopter  replacement,  for  instance,  could  easily 
be  incorporating  a  premature  decision  that  the  underlying  capability  need  is  best  met 
by  another  helicopter  of  some  sort. 

The  danger  can  arise  not  only  from  an  inappropriate  focus  on  existing  C/CSS 
(for  example,  the  notion  that  a  particular  platform  must  be  replaced  by  a  similar  type 
of  platform),  but  also  a  focus  on  existing  concepts  of  operation  (CONOPS). 

Consider,  for  example,  a  sensor  that  provides  input  to  step  1  of  a  two-step  battlefield 
decision-making  process.  An  AoA  that  focuses  exclusively  on  how  well  the 
alternatives  support  step  1  could  easily  be  scoped  improperly,  as  illustrated  in 
Figure  2.  Here,  instead  of  a  helicopter  replacement  AoA,  we  have  what  might  be 
termed  the  “Step  1  Replacement  AoA.”  In  this  example,  Alternative  A  would  certainly 
be  included  in  the  AoA,  since  it  improves  the  effectiveness  of  step  1  relative  to  the 
base  case.  However,  Alternative  B  might  very  well  be  excluded  unless  the  AoA 
scope  was  widened  to  account  for  the  potential  improvement  it  could  bring  to  step  2. 


Effectiveness 

"Step  1  Replacement' AoA 

Base  Case 

Alternative  A  Alternative  B 

- 

“Ti  in 

1 

V 

Step  1 

0.80 

0.90 

0.75 

. t . 

Step  2 

0.60 

[no  change] 

0.95 

1 

Overall 

Outcome* 

0.48 

0.54 

0.71 

‘Overall  effectiveness  =  effectiveness  of  Step  1  x  effectiveness  of  Step  2 

Figure  2.  Example  of  Inappropriate  Focus  on  Existing  CONOPS 
Inappropriate  Focus  on  a  Single  Warfighting  Domain 

By  the  time  an  AoA  begins,  a  DoD  lead  component  (most  often,  one  of  the 
services)  has  been  designated  and  has  been  given  responsibility  for  conducting  the 
analysis.  That  component  will  have  played  a  key  role  in  analyzing  the  capability 
need  and  developing  the  ICD;  it  will  also  have  had  to  identify  a  possible  range  of 
feasible  alternatives  in  support  of  the  MDD.20  There  are  many  reasons  behind  the 
designation  of  DoD  lead  component:  for  example,  the  availability  of  technology 
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development  expertise,  personnel,  models  and  simulations,  and  budget.  Often,  the 
nature  of  the  capability  need  documented  in  the  ICD  makes  the  choice  of  a  lead 
service  “obvious.”21  To  explore  this  phenomenon  further  would  require  a  detailed 
look  at  the  role  of  the  services  in  the  various  processes  that  provide  input  to  the 
definition  of  capability  gaps  and  needs22 — an  assessment  well  beyond  the  bounds  of 
this  study. 

Regardless  of  the  reasons,  service  proponency  of  AoAs  was  identified  by 
several  interviewees  as  a  potential  contributor  to  overly  narrow  scoping.  Colloquially 
speaking,  “If  the  Navy  is  conducting  the  AoA,  the  solution  is  probably  going  to  live  in 
the  water;  if  the  Air  Force  is  conducting  the  AoA,  the  solution  is  likely  to  have  wings; 
etc.”  The  problem  is  not  necessarily  that  one  service  deliberately  sets  out  to  exclude 
solutions  that  could  be  developed  by  another:  each  particular  service  simply  thinks 
about  capability  needs  and  their  solutions  in  a  particular  way,  corresponding  to  the 
warfighting  domain  it  represents.  Such  viewpoints  may  not  lend  themselves  well  to 
the  full  exploration  of  solutions  to  joint  or  coalition  warfighting  needs. 

Inclusion  of  Extraneous  Issues 

The  national  security  landscape  is  such  that  new  concepts  and  ideas 
regularly  burst  into  the  universe  of  discourse:  consider,  for  instance,  the  history  of 
low-intensity  conflict,  information  operations,  or  supply  chain  management.  For 
purposes  of  this  report,  the  risk  is  that  a  given  concept  or  issue  may  be  so  much  in 
vogue  that  AoA  guidance  mandates  that  it  be  considered — and  therefore  that  an 
alternative  be  included — simply  because  it  is  a  hot  topic. 

Lack  of  Agility  in  Governance  and  Management  Processes 

As  noted  in  the  first  section,  a  common  complaint  is  that  both  AoAs  and  the 
JCIDS  process  take  too  long.  Unfortunately,  the  result  is  often  a  reluctance  to  make 
adjustments  during  AoA  execution.  For  example,  even  though  interim  AoA  results 
might  suggest  that  the  underlying  capability  need  should  be  reexamined  or 
articulated  differently,  this  seldom  occurs  because  “too  much  time  has  already  been 
expended  just  to  get  to  this  point.”  One  possible  result:  AoAs  that  are  improperly 
scoped  because  they  ignore  fact-of-life  changes  that  have  rendered  some 
alternatives  obsolete  or  made  others  feasible. 

An  associated  problem  is  that  governance  processes  may  not  be  able  to  keep 
pace  with  the  need  for  change.  Suppose  an  AoA  study  team  is  able  to  determine, 
during  the  course  of  an  AoA,  that  the  costs  and  benefits  of  a  given  alternative, 

21  The  WSARA  stipulates  an  OSD  check  on  the  extent  to  which  requirements  approved  by  the  Joint 
Requirements  Oversight  Council  (JROC)  have  “engaged  in  consideration  of  issues  of  joint  portfolio 
management,”  but  notes  that  this  was  already  required  by  DoD  instruction.  Of.  Public  Law  1 11-23, 
§201  (c)(3). 

2  See,  for  example,  Frittman,  McGee,  &  Yuhas  (2013). 
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mandated  by  study  guidance,  will  almost  certainly  cause  it  to  be  dominated  by  the 
others.  In  theory,  a  SAG  could  grant  a  waiver  or  change  of  scope.  However,  if  the 
approval  chain  leading  to  the  SAG  is  such  that  four  months’  worth  of  justification  is 
required  to  avoid  three  months  of  wasted  effort,  the  change  is  unlikely  to  happen. 

Applying  a  Systems  View 

Systems  Thinking  and  the  Conceptagon 

We  used  a  soft  systems  framework,  the  Conceptagon,23  to  derive 
comparisons  and  characterizations  that  helped  us  better  understand  the  factors  that 
affect  AoA  scoping.  The  Conceptagon  framework  is  organized  around  seven  triplets 
(cf.  Figure  3)  of  system  attributes. 

We  concentrated  on  four  of  these  triplets:24 

•  Wholes,  Relationships,  Parts.  The  identification  of  the  system  at  hand, 
the  constituent  pieces,  and  the  relationships  that  bind  those  pieces. 

•  Structure,  Process,  Function.  The  composition,  arrangement,  or 
organization  (structures)  a  system  employs  to  support  the  key  activities 
(processes)  necessary  to  produce  the  desired  system  behavior 
(function). 

•  Inputs,  Transformations,  Outputs.  The  items  that  enter  the  system 
(inputs)  and  exit  it  as  products  or  deliverables  (outputs)  and  the 
changes  (transformations)  that  convert  inputs  to  outputs. 

•  Interior,  Boundary,  Exterior.  The  perimeter  that  separates  entities  that 
comprise  the  system  from  entities  outside  its  control. 


23  Boardman  &  Sauser,  2008. 

24  The  remaining  triplets  (notably,  Command,  Control,  Communication;  and  Openness,  Hierarchy, 
Emergence)  are  better  suited  to  identifying  organizational  or  governance  solutions,  which  are  not  the 
focus  of  this  paper. 
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Figure  3.  Conceptagon  Framework 

(Boardman  &  Sauser,  2008) 


Analysis  of  Selected  Conceptagon  Triplets 

Wholes,  Relationships,  Parts 

What  system  are  we  analyzing  when  considering  the  problem  of  AoA  scope? 
Figure  4  identifies  two  very  different  kinds  of  systems  at  work.  The  upper  portion  of 
the  diagram  depicts  what  might  be  termed  the  AoA  Management  and  Execution 
System.  That  system  comprises  the  organizational  entities  responsible  for  planning, 
conducting,  documenting,  and  applying  the  AoA.  The  lower  portion  of  the  diagram 
depicts  an  entirely  different  kind  of  system.  Specifically,  the  label  AoA  Object  System 
describes  the  set  of  C/CSS(s)  that  are  the  object(s)  of  study  in  the  AoA. 

In  Figure  4,  the  focus  is  on  the  period  that  precedes  the  AoA  itself.  During  this 
timeframe,  the  Management  and  Execution  System  produces  several  outputs  that 
impact  the  scope  of  the  ensuing  AoA:  documentation  of  the  underlying  capability 
need  (in  the  form  of  an  ICD),  documentation  of  the  MDD  (via  an  Acquisition  Decision 
Memorandum25),  and  formal  AoA  Study  Guidance.  All  these  outputs  are  based  on 
the  assessment  of  current  and  programmed  C/CSSs,  existing  CONOPS  (not 
shown),  and  the  resulting  mission  outcomes.  The  capability  gap  documented  in  the 
ICD  is  associated  with  the  particular  set  of  C/CSSs  that  should  have  been  capable  of 


25  MDD  documentation  can  have  an  important  bearing  on  AoA  scope:  USD(AT&L)  (2010)  required 
that  the  MDD  be  based  on  evidence  of  a  range  of  “candidate  materiel  solution  approaches  [that]  have 
the  potential  to  effectively  address  the  capability  gap(s)  ...”  and  are  technically  feasible. 
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producing  the  required  capability.  This  set  will  become  the  baseline  alternative  in  the 
AoA.  The  designation  of  the  baseline  establishes  a  boundary  for  the  Object  System. 

Figure  5  depicts  AoA  execution.  Here,  the  output  of  the  Management  and 
Execution  System  is  the  completed  AoA.  Each  alternative  represents  a  different 
possible  instantiation  of  the  Object  System  and  is  assessed  based  on  its  outputs: 
namely,  the  predicted  mission  outcomes  it  would  produce  or  enable.  The 
assessment  may  deem  some  alternatives  infeasible  on  the  basis  of  these  outcomes; 
it  may  also  help  identify  other  alternatives  that  should  be  considered.  Notice  that  the 
reconfigurations  that  give  rise  to  the  alternatives  may  change  the  definition  of  the 
boundary:  A  C/CSS  that  was  formerly  outside  the  boundary  (for  example,  the 
helicopter  at  the  bottom  right  of  the  collage  in  Alternatives  1  and  2)  may  find  itself 
inside  the  boundary  (the  same  helicopter,  dropping  a  rectangular  object  in 
Alternative  3).  Thus,  the  AoA  scope  is  the  envelope  of  the  boundaries  formed  by  all 
the  instantiations  of  the  Object  System  (that  is,  all  the  alternatives). 
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Figure  4.  The  Two  AoA  Systems  (Prior  to  AoA  Execution) 
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Figure  5.  The  Two  AoA  Systems  (During  AoA  Execution) 


In  summary,  the  AoA  scope  is  a  property  of  a  system  output  (because  an  AoA 
is  an  output  of  the  AoA  Management  and  Execution  System).  It  results  from  one 
system  (that  is,  the  Management  and  Execution  System)  making  decisions  about  the 
boundary  of  another  (that  is,  the  Object  System)  both  before  and  during  the  analysis. 
Thinking  of  scope  in  this  way  allows  us  to  introduce  several  other  principles  of 
systems  analysis  (see  the  last  section  of  this  paper). 

Structure,  Process,  Function 

An  AoA  lies  at  the  intersection  of  the  three  principal  DoD  decision  support 
systems  (JCIDS;  the  Planning,  Programming,  Budgeting,  and  Execution  System 
[PPBES];  and  the  DAS).  Each  of  the  processes  that  govern  those  three  systems 
impacts  an  AoA — and,  in  particular,  the  actions  taken  by  the  elements  of  the  AoA 
Management  and  Execution  System  to  determine  its  scope.  Figure  6  summarizes 
the  key  relationships. 

In  some  cases,  it  is  fairly  obvious  how  these  functions  impact  decisions 
regarding  the  scope  of  an  AoA.  For  example,  resource  availability  can  limit  AoA 
scope  directly  (the  fewer  the  resources  available  for  AoA  execution,  the  fewer  the 
alternatives  that  can  be  analyzed  to  a  given  level  of  detail)  or  indirectly  (the  fewer  the 
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resources  available  for  solutions  to  capability  needs,  the  fewer  the  alternatives  that 
will  be  deemed  affordable).  Less  obvious  are  some  of  the  relationships  we  noted  in 
our  discussion  of  why  some  AoAs  are  poorly  scoped — for  example,  the  fact  that  the 
set  of  alternatives  is  profoundly  shaped  by  the  way  in  which  a  capability  gap  is 
articulated.  This  latter  point  is  taken  up  in  more  detail  in  the  last  section. 


Process* 

Function  (relative  to  AoAs) 

Key  Structural 

Elements 

JCIDS 

Assess  current  and  programmed  forces 
Identify  capability  gaps  and  needs 

Joint  Staff,  JROC 

Combatant  commands 
Component  using  commands 

PPBES 

Determine  available  resources  /  affordability 
Enable  capability  development 

OSD  CAPE 

USD(Comptroller) 

Component  HQ 

DAS 

Acquire  future  forces 

Develop  future  capabilities 

USD(AT&L) 

Component  AEs  and 
acquisition  commands 

*The  entries  in  this  column  are  all  termed  “systems”;  however,  the  shorthand  reference  is  to  the  processes 
that  govern  the  operation  of  these  systems. 


AE  =  Acquisition  Executive  OSD  CAPE  =  Office  of  the  Secretary  of  Defense,  Cost  and  Program  Evaluation 

DAS  =  Defense  Acquisition  System  PPBES  =  Planning,  Programming,  Budgeting,  and  Execution  System 

JCIDS  =  Joint  Capabilities  Integration  and  Development  System  USD(Comptroller)  =  Under  Secretary  of  Defense  (Comptroller) 

JROC  =  Joint  Requirements  Oversight  Council  USD(AT&L)  =  Under  Secretary  of  Defense  (Acquisition  Technology  &  Logistics) 


Figure  6.  DoD  Decision  Support  Functions  That  Impact  AoA  Scope 
Inputs,  Transformations,  Outputs 

Figure  7  depicts  the  notion  that  AoAs  transform  information  about  forces, 
capabilities,  and  C/CSSs  into  information  that  aids  decision-making  by 
characterizing  the  trade  space  of  alternative  solutions.  This  view  extends  the 
previous  triplet  by  identifying  additional  factors  that  impact  AoA  scope.  If  data  and 
models  are  not  in  place,  for  example,  the  resulting  inability  to  analyze  certain 
alternatives  within  time  and  resource  constraints  could  act  to  inappropriately  limit 
scope. 
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Figure  7.  AoA  Inputs,  Constraints,  Outputs,  Enablers 
Interior,  Boundary,  Exterior 

The  concept  of  AoA  scope  as  a  boundary  on  the  Object  System  was 
discussed  earlier  in  this  section.  The  earlier  discussion  showed  the  boundary  in  what 
might  be  termed  C/CSS  space.  However,  this  boundary  has  other  dimensions  as 
well — for  example,  temporal  dimensions.  Figure  8  shows  that  some  actions  taken 
before  AoA  initiation  (those  on  the  left  side  of  the  figure)  generally  have  broad 
impacts  on  AoA  scope  (for  example,  by  excluding  whole  classes  of  technologies  or 
solution  approaches).  Actions  taken  during  AoA  execution  tend  to  impact  AoA  scope 
more  narrowly  (for  example,  making  exclusions  within  allowable  classes). 
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Before  AoA 

During  AoA 

Identification  of  capability  gaps 

Gap  assessment — CBA 

Gap  documentation — ICD 

Material  Development  Decision— ADM 
AoA  Guidance 

Study  plan  development 

Characterization  of  alternatives 

Screening  of  alternatives 

Final  selection  of  alternatives 

CBA  =  Capabilities-Based  Assessment 

ICD  =  Initial  Capabilities  Document 

ADM  =  Acquisition  Decision  Memorandum 

Figure  8.  Timeframe  of  Actions  That  Impact  AoA  Scope 

Recommendations  and  Observations 

We  have  combined  insights  based  on  the  system  formulation  of  the  problem 
with  insights  gained  during  interviews  to  arrive  at  a  series  of  recommendations, 
which  are  presented  below,  along  with  some  concluding  observations. 

Recommendations 

The  guiding  principles  that  follow  are  intended  to  help  participants  in  the  AoA 
Management  and  Execution  System  reduce  the  incidence  of  improperly  scoped 
AoAs. 

Focus  on  Outputs  and  Think  Backwards 

An  alternative  is  any  potential  configuration  of  C/CSSs  that  could  produce  the 
required  output(s):  namely,  the  military  outcomes  that  would  address  the 
documented  capability  gap  to  some  degree.  In  formulating  alternatives,  it  is  tempting 
to  think  forwards — that  is,  to  envision  a  particular  set  of  C/CSSs  and  determine 
whether  it  produces  the  required  outcome(s)  (or  could  be  modified  or  reconfigured  to 
do  so). 

To  some  extent,  the  JCIDS  process  requires  participants  to  think  in  terms  of 
particular  C/CSSs  during  the  formulation  and  analysis  of  capability  gaps.  Sponsors 
of  capability  gaps  are  directed  to  consider  whether  “capability  solutions  which  can 
satisfy  the  Sponsor  capability  requirements  exist  elsewhere  in  the  Joint  force.”26  And 
CBAs  require  “the  operational  assessment  of  the  current  and  programmed  force.’’27 

Consideration  of  C/CSSs  during  gap  formulation  is  actually  inevitable:  It  may 
be  possible  to  think  about  a  capability  without  reference  to  the  system  that  provides 


26  Chairman,  Joint  Chiefs  of  Staff,  2012,  Enclosure  A,  paragraph  5.b  and  Figure  A-2  (emphasis 
added). 

27  Ibid.,  Appendix  B  to  Enclosure  A,  CBA  Guide ;  paragraph  2.e  (emphasis  added). 
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that  capability;  however,  it  is  not  possible  to  think  about  a  capability  gap  without 
reference  to  one  or  more  sets  of  C/CSSs  that  fails  to  do  so.  Thus,  by  the  time  an 
MDD  is  reached  and  an  ICD  generated,  the  Object  System — including  its  boundaries 
and  interfaces — has  already  been  conceived  ...  with  reference  to  the  current  and 
programmed  force.  The  danger  is  that  this  process  may  induce  tunnel  vision  and 
drive  the  AoA  to  an  overly  narrow  scope. 

How  to  escape  this  dilemma?  At  some  point  before  the  formulation  of  AoA 
study  guidance,  there  should  be  an  attempt  to  work  backward  from  a  (C/CSS- 
neutral)  statement  of  the  required  capability  to  a  set  of  feasible  alternatives  that 
could  potentially  satisfy  it.  Such  an  attempt  should  not  be  bound  in  any  way  by  the 
thinking  contained  in  the  CBA. 

Formal  systems  analysis  tools  and  methods  can  help  in  drawing  the 
necessary  linkages.  These  methods  also  show  clearly  the  difference  between  the 
forward  and  backward  thought  processes  alluded  to  above.  Within  version  2.0  of  the 
DoD  Architecture  Framework  (DoDAF),  for  example,  the  “forward”  progression  of 
thought  corresponds  to  the  movement  from  Operational  View  (OV)  to  Capability 
View  (CV).28  It  asks,  “What  is  the  CV  associated  with  this  particular  OV?”  This 
process  may  be  used  during  the  assessment  of  current  and  programmed  forces  to 
identify  capability  gaps,  resulting  in  an  ICD. 

What  we  have  termed  the  backward  thought  progression  can  be  represented 
by  a  subsequent  movement  from  CV-1  (Capability  Vision)  to  CV-6  (Capability  to 
Operational  Activities  Mapping),  as  shown  in  Figure  9.  The  CV-1  essentially 
captures  the  sense  of  the  ICD.  Forcing  the  analyst  to  carefully  translate  capabilities 
to  activities,  it  becomes  possible  again  to  work  from  a  clean  sheet  of  paper  by  asking 
questions  such  as  “Are  there  other  kinds  of  activities  or  combinations  of  activities 
that  could  produce  these  capabilities?”  “How  might  such  activities  be  performed  or 
linked  to  do  that?”  In  the  end,  one  is  asking,  in  DoDAF  terms,  “What  kinds  of  OVs 
could  correspond  to  this  CV?”  Or,  in  other  words,  “What  kinds  of  C/CSSs,  interacting 
with  each  other  and  with  the  user  in  what  ways,  could  potentially  meet  the  stated 
need?” 


28  Note  that  DoDAF  itself  does  not  prescribe  a  progression  of  thought.  In  applying  DoDAF,  one  can 
start  almost  anywhere,  producing  only  the  views  required  for  a  particular  purpose. 
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Figure  9.  Translating  Capabilities  to  Activities:  Using  DoDAF  v2.0 

(Wayson,  2010,  Slides  9  and  14)29 


The  backwards  thought  transition  from  CV  to  OV  has  an  additional  benefit.  By 
setting  up  the  question  of  user  interaction  with  the  system,  it  stimulates  additional 
thinking  about  the  Object  System  boundary  and  how  that  boundary  contributes  to 
the  definition  of  alternatives  (see  the  next  subsection). 

Start  With  the  Exterior  and  Work  Inwards 

The  usual  thought  process  is  to  start  with  a  collection  of  C/CSSs  believed  to 
represent  an  instantiation  of  the  Object  System  and  move  conceptually  outwards. 
Those  entities  without  which  the  Object  System  could  not  exist  or  function — provided 
they  are  within  the  decision  maker’s  sphere  of  influence  and  authority — are  deemed 
to  lie  in  the  interior;  all  other  entities  are  in  the  exterior.  Although  this  may  be  a 
sound  method  of  refining  an  initial  definition,  it  may  not  be  the  best  approach  to 
identifying  alternative  instantiations  of  the  Object  System. 

The  suggested  approach  is  to  reverse  the  process — that  is,  to  start  by 
identifying  entities  that  are  clearly  not  part  of  the  system  and  then  moving 
conceptually  inwards.  If  the  stated  need  is  to  improve  the  autonomous  navigation 
capabilities  of  an  existing  fleet  of  aircraft,  for  example,  it  may  be  easy  to  determine 
that  alternatives  involving  entirely  new  airframes  are  unaffordable.  The  airframes 
themselves  now  lie  in  the  exterior  of  the  Object  System  and  impose  a  set  of 
compatibility  constraints.  However,  it  may  not  be  clear  whether  alternatives  that 
upgrade  both  navigation  and  fire  control  systems  are  cost-effective.  The  outside-in 


29  In  this  example,  the  CV-6  includes  capabilities  not  shown  in  the  CV-1 .  The  additional  capabilities 
implicit  in  the  CV-1  were  formally  identified  in  the  Capability  Taxonomy  (CV-2). 
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approach  might  allow  both  types  of  solutions  among  the  potential  alternatives  in  an 
AoA.30 

Outside-in  exploration  of  the  boundary  also  helps  identify  interfaces  that 
“[participate]  in  constituting  the  system.”31  The  eardrum,  for  example,  “forms  the 
boundary  between  the  inner  and  the  outer  ear,  but  at  the  same  time  it  exists  in  order 
to  let  the  sound  waves  through.”32  In  the  same  way,  the  interface  constraints 
identified  during  an  outside-in  exploration  of  the  Object  System  boundary  point 
toward  important  characteristics  of  AoA  alternatives.  They  can  represent  more  than 
simply  feasibility  constraints.  For  example,  they  can  uncover  important  measures  of 
effectiveness. 

Apply  Constraints  Carefully 

Every  constraint  imposed  or  accepted  constitutes  a  decision  to  exclude  some 
portion  of  the  solution  trade  space  from  further  consideration.  Accordingly,  it  should 
be  supported  by  some  testable  rationale,  that  is,  an  explanation  of  what  portion  of 
the  trade  space  is  being  excluded  and  why  it  is  not  worth  considering.  An  important 
question  to  ask:  What  would  it  take  to  change  this  constraint?  To  see  the  importance 
of  asking  this  question,  consider  several  examples. 

The  first  example  considers  the  well-known  constraint  termed  affordability. 

For  many  years,  decision  makers  relied  on  AoAs  to  identify  cost-effective  solution 
approaches,  only  to  discover  that  the  resulting  programs  could  not  be  executed  with 
available  resources — that  is,  they  were  unaffordable.  In  response,  the  USD(AT&L) 
has  mandated  that  affordability  be  considered  at  Milestone  A.33  An  overly  restrictive 
approach  would  be  to  define  affordability  as  the  difference  between  expected  out- 
year  obligational  authority  and  the  expected  out-year  costs  of  current  and 
programmed  C/CSSs.  That  approach,  however,  fails  to  consider  the  possibility  that 
the  benefits  of  meeting  the  capability  need  at  hand  may  be  so  great  as  to  be  worth 
terminating  or  forgoing  some  current  or  programmed  C/CSSs.  Rather  than  unduly 
constrict  the  trade  space  with  this  narrow  definition,  decision  makers  would  be  better 
served  by  asking,  “What  other  capabilities  would  we  be  willing  to  forgo  to  achieve 
this  one?”34 

Another  example  was  mentioned  earlier:  The  Object  System  is  usually 
constrained  to  exclude  elements  that  are  outside  the  decision  maker’s  sphere  of 


30  An  interesting  parallel  in  the  world  of  object-oriented  software  design  is  the  recent  interest  in  what 
is  now  being  termed  outside-in  development.  For  a  practical  illustration,  see  Bache  (2013). 

31  Cilliers,  2005,  p.  611. 

32  Ibid. 

33  Husband  and  Kaspersen,  2012,  p.  9. 

34  Based  on  the  interviews  we  conducted,  there  is  evidence  that  this  question  is  now  being  posed  as 
a  way  of  attaching  resource  priority  levels  to  DoD  capability  needs. 
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influence  or  authority.35  Suppose  the  decision-making  authority  is  organization  X  and 
all  materiel  alternatives  are  believed  to  be  related  to  technology  A,  for  which  that 
organization  is  responsible.  Further,  suppose  that  the  AoA  study  team  discovers, 
while  developing  its  study  plan,  a  set  of  potential  alternatives  based  on  technology 
B,  for  which  the  responsible  organization  is  Y  (a  parallel  organization  in  the 
hierarchy).  The  argument  that  “those  aren’t  our  technologies  and  we  can’t  make 
decisions  about  them”  should  not  be  considered  sufficient  grounds  for  dismissing  the 
newly  discovered  set  of  alternatives — even  though  the  consequences,  in  terms  of 
process,  may  be  painful  (for  example,  moving  the  decision-making  authority  to  a 
higher  level). 

Even  the  capability  gap  itself  can  be  articulated  in  such  a  way  as  to  over¬ 
constrain  the  solution  trade  space — as,  for  example,  might  occur  if  alternatives  that 
fail  to  close  the  gap  completely  are  dropped.  During  the  interviews  we  conducted, 
we  learned  that  new  JCIDS  guidance  under  development  will  address  this  point  by 
stating  clearly  that  ICDs  should  not  include  threshold  values  for  key  performance 
parameters. 

Iterate  and  Reduce  Uncertainty 

Because  the  environment  can  change  over  time,  the  boundary  of  the  Object 
System  should  be  allowed  to  change  as  well.  Standard  AoA  practice  accounts  for 
such  change  through  the  normal  screening  process.  Less  common,  but  equally 
important,  is  the  occasional  need  to  add  alternatives  or  fundamentally  change  the 
nature  of  the  alternative  set.  Such  changes  can  arise  for  a  number  of  reasons:  For 
example,  during  the  interval  between  the  CBA  and  the  AoA,  the  state  of  technology 
may  have  evolved  such  that  alternatives  once  thought  to  be  infeasible  are  now 
feasible. 

The  recommendation  is  to  take  an  iterative  approach,  specifically,  to  check 
and  recheck  the  validity  of  assumptions  and  constraints  throughout  AoA  preparation 
and  execution.  Note  that  this  concept  extends  to  the  period  before  the  AoA  has 
actually  begun.  In  the  earlier  example  of  the  aircraft  navigation  system  (see  the 
subsection  titled  Start  With  the  Exterior),  we  remarked  that  a  certain  decision  might 
be  easy  to  determine  and  that  another  decision  might  result  in  a  certain  outcome. 
Ideally,  these  decisions  are  informed  by  quick  analysis  at  a  low  level  of  detail. 
Although  the  analysis  results  may  come  with  a  wide  range  of  uncertainty,  they  may 
still  be  sufficient  to  allow  for  the  right  decision  regarding  the  Object  System 
boundary.  If  not,  the  analysis  is  refined  to  the  point  that  it  does  support  the  decision. 

The  associated  need  is  for  CBA  and  AoA  governance  processes  that  can 
respond  quickly  to  the  need  for  such  adjustments.  The  establishment  of  the  SAG 


35  Cf.  Oliveira,  1973,  p.  3. 
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(supported  by  Joint  Staff  Functional  Control  Boards)  as  an  ongoing  governance 
mechanism  is  undoubtedly  a  step  in  the  right  direction — although  there  is  evidence 
from  our  interviews  that  the  process  is  not  yet  as  agile  as  it  needs  to  be.  For 
example,  it  is  not  clear  to  what  extent  the  SAG  can,  on  its  own  authority,  redirect  the 
execution  of  an  AoA  along  lines  that  deviate  from  the  boundaries  established  by  a 
validated  ICD.  If  the  impacts  of  real-world  change  can  be  accommodated  only  by 
restarting  the  JCIDS  process  from  scratch,  they  are  unlikely  to  be  considered  at  all. 

Summary  and  Conclusions 

A  scheme  for  applying  these  recommendations  is  summarized  in  Figure  10, 
which  shows  the  alignment  of  key  principles  according  to  stakeholder  roles  and 
process  timelines.  The  absence  of  an  entry  does  not  mean  that  participant  is  exempt 
from  that  particular  principle — merely  that  it  is  not  necessarily  key  to  AoA  scoping  at 
that  particular  point  in  the  AoA  process. 


When 
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Identification  of 
Capability  Gap 

Development  of 
AoA  Guidance 

Execution  of 

AoA 

Post-AoA  (Review 
/  Implementation) 

AoA  Decisionmakers 
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00  | 

0 
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000 

00 

00 

DoD  Process  Owners 

000 

(  1 A  jj  2 

00 

00 

AoA  Study  Team 

Acquirers,  Developers, 
Labs 

^  0  ' 

(  j/)  ^  ^  ^ 

Focus  on  Outcomes  and  Think  Backwards 

^ _ ^  Apply  Constraints  Carefully 

Start  with  the  Exterior  and  Work  Inwards 

Iterate  and  Reduce  Uncertainty 

Figure  10.  Summary  of  Key  Principles  for  Improving  AoA  Scoping 

One  important  implication  of  Figure  10  is  that  systems  thinking  should  not  be 
considered  an  exercise  solely  for  AoA  study  team  analysts.  Senior  executives, 
warfighters,  and  DoD  business  process  owners  can  all  benefit  from  applying  a 
systems  view  to  the  practice  of  AoAs.  That  practice  continues  to  evolve  within  the 
DoD,  and  the  challenges  of  the  future  will  doubtless  be  different  from  those  of  the 
past.  Cultural,  organizational,  and  individual  behaviors  will  continue  to  frustrate  the 
best  process  designs.  Nevertheless,  we  are  confident  that  the  above  principles  can 
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improve  the  DAS  by  reducing  the  incidence  of  poorly  scoped  AoAs  and  thus  enable 
better  acquisition  outcomes. 
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